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FIELD OF INVENTION 

[0001] This invention relates to software for making a panoramic image on a low profile system. 
DESCRIPTION OF RELATED ART 

[0002] Digital photography is becoming more popular today as digital cameras and scanners are 
becoming widely available. Digital images can be created either by capturing a scene using a 
digital camera or digitizing a traditional film-based photograph using a scanner. One particular 
advantage of digital photography over traditional film-based photography is that digital images 
can be easily manipulated or edited for better presentation. 

[0003] When a photographer captures a scene using a camera, the desired field of view may be 
larger than the normal field of view of the camera. Digital photography allows a panoramic 
image to be produced without the need of purchasing special equipment such as a panoramic 
camera or a fisheye lens. For example, a photographer with a digital camera may capture a 
series of digital pictures of a scene by rotating the camera and taking pictures in a sequence of 
different directions. The captured images may then be projected onto a cylinder and then 
stitched together to produce a panoramic picture of the scene. Similarly, film-based photographs 
can be digitized, and the panoramic picture can be composed by projecting and stitching together 
the digitized images. Presently, digital image programs are available for stitching muhiple 
digital images together to form a panoramic picture. Exemplary programs include Ulead Cool 
360^^, Live Picture PhotoVista™, and MGI PhotoSuite III™. 

[0004] Typically a digital image program is executed by a personal computer, which has 
sufficient processor power and memory to buffer and manipulate the series of pictures to be 
stitched into the panoramic picture. Typically the resulting panoramic picture is saved as a JPEG 
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image. The JPEG image is saved horizontally scan line by scan line and thus spans the width of 
the entire series of pictures. Thus, the personal computer must buffer the entire series of 
pictures, decide how to stitch them, and then write the resulting panoramic picture horizontally 
scan line by scan line. 

[0005] As cellular phones and handhelds (e.g., a Palm devices) with built-in cameras become 
increasing popular, these devices can be expected to create panoramic pictures despite their slow 
processors and limited memories. Similarly, digital cameras can be expected to perform 
panoramic functions. Thus, what is needed is panoramic software for portable devices that 
efficiently utilizes their limited resources. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0006] Fig. 1 is a flowchart of a method for creating a panoramic image in one embodiment of 
the invention. 

[0007] Fig. 2 illustrates the creation of a panoramic image using the method of Fig. 1 in one 
embodiment of the invention. 

[0008] Figs. 3 and 4 illustrate the projection of an image onto a cylinder for creating a panoramic 
image in one embodiment of the invention. 

[0009] Fig. 5 illustrates a resolution pyramid used for matching two images in one embodiment 
of the invention. 

[0010] Fig. 6 illustrates a minimum color difference path in an overlapping region between two 
images in one embodiment of the invention. 

[0011] Fig. 7 illustrates a blending operation to smooth the transition between two images in the 
panoramic image in one embodiment of the invention. 

[0012] Use of the same reference mmibers in different figures indicates similar or identical 
elements. 
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SUMMARY 

[0013] In one embodiment of the invention, a method for generating a panoramic image includes 
receiving a first image, dividing the first image into a first portion and a second portion, rotating 
the first portion of the first image, saving the rotated first portion of the first image in a 
nonvolatile memory, receiving a second image, dividing the second image into a third portion 
and a fourth portion, matching an overlapping region between the second portion of the first 
image and the third portion of the second image, stitching the second portion of the first image 
and the third portion of the second image to form a first stitched image, rotating the first stitched 
image, and saving the first stitched image in the nonvolatile memory. 

DETAILED DESCRIPTION 

Method for Creating a Panoramic Image 

[0014] Fig. 1 is a flowchart of a method 10 for creating a panoramic image in one embodiment 
of the invention. Method 10 can be implemented with software executed by hardware on a 
portable device such as a camera phone, a handheld device, or a digital camera. 

[0015] In step 12, the device receives or captures an image (e.g., image 1 in Fig. 2) in a series of 
images that makes up a panoramic image (e.g., panoramic image 8 in Fig. 2). 

[0016] In step 14, the device projects the current image (e.g., image 1) onto a cylinder to 
generate a warped image. The warped image presents a realistic panoramic view to the user by 
placing the user at the center of the cylinder with the series of images projected onto the wall of 
the cylinder. 

[0017] In step 16, the device divides the current image (e.g., image 1) into a left portion (e.g., 
left portion 1 A in Fig. 2) and a right portion (e.g., right portion IB in Fig. 2). The device then 
orthogonally rotates the left portion of the current image (e.g., left portion 1 A of image 1) in a 
.first direction (e.g., clockwise), and saves the rotated left portion as the first part of the 
panoramic image (e.g., panoramic image 8) in nonvolatile memory. In one embodiment, the 
panoramic image is saved in JPEG format so the rotated left portion is processed and saved 
horizontally scan line by scan line. The right portion of the current image (e.g., right portion IB 
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in image 1) is not yet rotated because it will be used to determine an overlap between the current 
image and the next image. 

[0018] In step 18, the device receives or captures the next image (e.g., image 2 in Fig. 2) in the 
series of images that makes up the panoramic image (e.g., panoramic image 8). In one 
embodiment, the viewfinder of the device displays the right portion of the previous image (e.g., 
right portion IB of image 1) so the user would know what portion of the scene should be 
included in the next image (e.g., image 2) to form the panoramic image. 

[0019] In step 20, the device projects the current image (e.g., image 2) onto the cylinder to 
generate another warped image. 

[0020] In step 22, the device divides the current image (e.g., image 2) into a left portion (e.g., 
left portion 2A in Fig. 2) and a right portion (e.g., right portion 2B in Fig. 2). The device then 
matches the right portion of the previous image (e.g., right portion IB of image 1) with the left 
portion of the current image (e.g., left portion 2A of image 2) to determine the overlap between 
the previous image and the current image. In one embodiment, the device only searches a sub- 
portion of the left portion of the current image (e.g., sub-portion 2A-1 of image 2 in Fig. 2) for a 
match with the right portion of the previous image (e.g., right portion IB of image 1). Once a 
match is foxmd, the device aligns and then stitches together the right portion of the previous 
image and the left portion of the current image to form a stitched image (e.g., stitched image 4 in 
Fig. 2). 

[0021] In step 24, the device blends the colors fi-om the right portion of the previous image (e.g., 
right portion IB of image 1) and the left portion of the current image (e.g., left portion 2A of 
image 2) to provide a smooth transition fi-om the previous image to the current image. 

[0022] In step 26, the device orthogonally rotates the stitched image in the first direction and 
then saves the rotated stitched image as a part of the panoramic image (e.g., panoramic image 8) 
in nonvolatile memory. As described above, in one embodiment, the panoramic image is saved 
in JPEG format so the rotated stitched image is processed and saved horizontally scan line by 
scan line. The right portion of the current image (e.g., right portion 2B of image 2) is not yet 



-4- 



rotated because it will be used to determine the overlapping region between the current image 
(e.g., image 2) and the next image (e.g., image 3 in Fig. 2). 

[0023] In step 28, the device determines if there is another image in the series of images that 
makes up the panoramic image. If so, step 28 is followed by step 18 and method 10 repeats until 
all the images in the series have been processed to form the panoramic image. If there is not 
another image in the series, then method 10 is followed by step 30. 

[0024] In step 30, the device orthogonally rotates the panoramic image (e.g., panoramic image 8) 
in a second direction (counterclockwise) so the panoramic image is now in the correct orientation 
for viewing by the user. The device can also crop out the curvature in the panoramic image to 
make the final image rectangular. The device then saves the final panoramic image in 
nonvolatile memory. 

Projecting an Image onto a Cylinder 

[0025] As described above in steps 14 and 20 (Fig. 2), the device projects the images that make 
up the panoramic image onto a cylinder as exemplified by images Ij and I2 in Fig. 3. This 
creates a realistic panoramic view to the user as if the user is standing at the center of the scene 
and viewing the surrounding when the user is actually viewing the series of images projected 
onto the wall of the cylinder. 

[0026] To project the images on the cylinder, several assumptions are made. First, the focal 
length of the camera is assumed to be fixed and known. For example, the focal length of the 
camera can be provided by the device manufacturer. Second, the camera is assumed to have no 
other motion other than rotation around while taking the series of images. Third, the rotational 
axis is assumed to be parallel to the y-axis of image. To simply the projection model shown in 
Fig. 3, it is assumed that the focal length of the camera is f, the radius of cylinder is f, and the 
distances from the rotational axis to the image planes are also f. 

[0027] Referring to Fig. 4, plane I is the image plane and cylinder Cy is the cylinder where the 
images are to be projected. Origin Oi is located at the center of image plane I and origin O is 
located at the center of cylinder Cy. Point Cj lies on the x-axis of image plane I and point C is 
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located at the intersection of line OCi and cylinder Cy. From trigonometry, the following 
correlation is inferred: 

—^ = arctan—^ (1) 
OOj 00 J 

[0028] Point Bj is an arbitrary point in image plane I. Point Ci is the projection of point Bi onto 
the X-axis on image plane L Line B'C is the projection of line BiCi onto cylinder Cy and point B 
is the projection of point Bi onto cylinder Cy. Here AOBC is similar to AOBiCi. From 
trigonometry, the following correlation is inferred: 

OC OCj OOj/cos%^ 

[0029] Suppose the image coordinates of point Bi is (x, y) and the cylinder coordinates of B is 
(x\ y'), equation 2 can be rewritten as: 

x! X 
X- f y 5 or x' = / arctan—^ and (3) 

j^ = /cosy,or y^ysQCy. (4) 



[0030] Thus, equations 3 and 4 are used to project the images that make up the panoramic image 
onto cylinder Cy. From equations 3 and 4, it can be seen that the relationship between;; and;; ' is 
linear if jc ' or x is fixed and the formula for x ' is independent of Thus, the calculation of jc ' and 
has been separated to reduce the complexity of the projection computation. 

[0031] For example, one vertical line having value x in the image plane corresponds to one 

projected vertical line having value x ' on the cylinder. Thus, value x ' only needs to be 

determined once for projecting one vertical line having value x ' in the image plane. As x ' is 

constant, values for each vertical line on the cylinder can be calculated simply as a function of 

x' X x' 

V because sec — is constant in equation 4. Values of arctan — and sec — can be stored in a 

/ f f 

lookup table instead of being calculated in real time to speed up the projection computation. 
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Matching Current and Previous Images 



[0032] As described above in step 22 (Fig. 1), the device must match the current image and the 
previous image in order to align and stitch the images together to form a panoramic image. This 
can be done by placing one image over the other and determining how closely the two images 
match. As described above, the overlapping region to be matched and searched can be limited to 
a portion of the total image area, such as left portion IB of image 1 (Fig. 2) and sub-portion 2A-1 
of image 2 (Fig. 2), to speed up the matching process and to reduce the computing cost. In one 
embodiment, the device searches for shared features between the two images and uses these 
shared features to align the images. Typical features used for matching includes points, lines, 
and region topology. In one embodiment, the device uses a conventional RANSAC (Random 
Sample Consensus) algorithm to match the shared features between the two images. 

[0033] To fiuther speed up the matching process and reduce the computing cost, the device 
utilizes a resolution pyramid in one embodiment of the invention. The use of the resolution 
pyramid is described in commonly owned U.S. Patent Application Serial No. 09/665,917, 
entitled "Image Matching Using Resolution Pyramids with Geometric Constraints," filed 
September 20, 2001, which is incorporated by reference in its entirety. The use of the resolution 
pyramid is briefly described in reference to Fig. 5. 

[0034] A resolution pyramid 50 can be constructed for each image to be matched. Resolution 
pyramid 50 includes n levels of the image at different resolutions, which range from the original 
resolution at level Lo to the coarsest resolution at level Lt. In one embodiment, each upper level 
is derived from a lower level down sampled 2 by 2 pixels to 1 pixel. 

[0035] At the top level, the image is small enough that matching two images can be 
accomplished relatively easily. For example, the device detects and matches features in the two 
images, or portions of the two images, at the top level to determine a camera motion. The device 
then uses the camera motion to guide the search at the next lower level. For example, the device 
first detects features in the previous image (or a portion of the previous image) at the next lower 
level. Instead of performing an exhaustive search of the current image (or a portion of the 
current image) for the shared features, the device searches areas of the current image (or a 
portion of the current image) where the camera motion predicts the shares features should be 
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located. After searching and matching the shared feature, the device determines a refined camera 
motion that can be used again to fiirther match the two images. 

Blending the Overlapping Region 

[0036] As described above in step 24 (Fig. 1), the overlapping region between two stitched 
images are blended to provide a smooth transition between images. In practice, the scene being 
captured often changes between the time when the device receives or captures the two images. 
This causes the content of the overlapped region between the two images to be less than 
identical. For example, if a man is walking in the scene, then the position of this man is different 
in these two images to be blended. A conventional blending method will result in a blurry 
panoramic image because the color values of the man would be blended into the image. 

[0037] Fig. 6 illustrates how the device prevents blurring in one embodiment of the invention. 
The device matches an overlapping region 84 between a right portion 8 IB of a previous image 
and a left portion of 82 A- 1 of the current image. To prevent blurring, the device first determines 
a color difference map of overlapping region 84 between the previous image and the current 
image. The device then determines a path 86 that has the minimum color difference in 
overlapping region 84. In one embodiment, a weighted color difference of a pixel is calculated 
as: 

Dij = d,j + d,_, j_, + d,_, j_, + d, j_, + d,,, j_j + d,,, j_, , (5) 

where Ay is the weighted color difference of a pixel (ij), dyis the color difference of pixel (ij) 
from the color difference map, and d^_2j^j,d^_j j_j,di j_j,di^j j^j, and d^^2j-i the color 

differences of the five lower neighbors of pixel (iJ) from the color difference map. Minimum 
color difference path 86 is the path that has the minimum color difference sum of all the pixels 
that make up the path out of all the possible paths through overlapping region 84. 

[0038] The device stitches the previous image and the current image by filling the left of 
minimum color difference path 86 with the previous image, and filling the right of minimum 
color difference path 86 with the current image. If the colors of a scan line on the two sides of 
path 86 are similar, then the device blends the color values of the two images along a blending 
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width W centered on path 86. The colors of a scan line on the two sides of path 86 are similar if 
the color difference of the pixel on path 86 in that scan line is less than a threshold value. If the 
colors of a scan line on the two sides of path 86 are too different, then the device does not blend 
the color values of the two images. Thus, the device prevents blurring by detecting a possible 
change in the scene and stops the blending of the two images. 

[0039] Fig. 7 illustrates how the device blends the colors of the two images on a scan line in one 
embodiment of the invention. As illustrated in Fig. 7, the color value of the previous (e.g., left) 
image on path 86 is greater than the current (e.g., right) image on path 86 for a particular scan 
line. Thus, color values are subtracted from the previous image along half of blending width W 
(i.e., W/2) while color values are added to the current image along blending width W/2 to 
provide a smooth transition between the two images along path 86. Conversely, if the color 
value of the previous image on path 86 is less than the color value of the current image on path 
86 for a particular scan line, then color values are added to the previous image along blending 
width W/2 and color values are subtracted from the current image along blending width W/2. 

[0040] First, the device determines color values that are to be added to or subtracted from the 
color values of the previous image and the current image along blending width W. The color 
value to be added or subtracted is the product of a curve 102 and the color difference dtj of the 
pixel on path 86. Specifically, the device uses the following formula: 



where C(x) is the color value to be added to or subtracted from a pixel located x away from pixel 



is the blending width. 

[0041] In one embodiment, to speed up the blending operation, the device defines blending 
width W to be the largest integer of 2" that is less than width Wmax of portion 8 IB and 82 A- 1 
(Fig. 6). By defining blending width W so, the device can use shift operation in equation 6 
instead of integral dividing to blend the color of the two images. 




(6) 



(i,j) on path 86, dy is the color difference of pixel 1 - 



X 



represents curve 102, and W 



W/2 
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[0042] Various other adaptations and combinations of features of the embodiments disclosed are 
within the scope of the invention. Numerous embodiments are encompassed by the following 
claims. 
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